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Abstract 



The New York Collaborative for Excellence in Teacher Preparation (NYCETP) is a 
multi-campus project designed to produce well-qualified teachers of science and mathematics. 
The Center for Advanced Study in Education (CASE) carried out a number of procedures for 
evaluation of NYCETP. On-going formative evaluation activities during the five years of the 
project included attending and documenting NYCETP meetings and conferences, providing 
feedback summaries on conferences and working groups, and evaluating the course conversion 
and course enhancement documents. One of the main goals of the evaluation was the design of 
activities that supported faculty development and intercampus faculty collaboration. These 
evaluation activities started with the use of case study methods in which a faculty member 
studied a course developed by faculty on a different campus. The process was expanded with 
the development and use of peer review procedures for course and curriculum materials 
designed for NYCETP. Another evaluation study expanded the peer review process of faculty 
course materials to the use of a lesson plan review form for education mathematics methods 
courses, with faculty rating teacher education students’ lesson plans. Survey questionnaires 
were developed and piloted to examine student, faculty, and administrator familiarity with 
NYCETP goals. 

A project sixth-year extension provided resources to participate in the NSF CETP 
Core Evaluation and Data Collection Project, to summarize the NYCETP data, and to plan for 
an integrated student data base and tracking system. These resources were also used to 
develop a three-year evaluation proposal that coordinates the NYCETP data collection goals 
with the CUNY-wide effort to develop a longitudinal data base for teacher education students 
on all campuses, including placement and follow-up information in line with both New York 
state and NCATE accreditation requirements. 
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I. Introduction and Procedures 



The New York Collaborative For Excellence in Teacher Preparation (NYCETP) was a 
project involving five campuses of the City University of New York (CUNY) and New York 
University (NYU). The goal of the Collaborative was stated as, to produce “well-qualified 
teachers of science and mathematics for New York City schools and to increase the number of 
individuals who enter and successfully complete teacher preparation requirements in science 
and mathematics.” The Collaborative efforts to meet this main objective were presented in six 
clusters of activities: (1) Rethinking college instruction — methodology and structures; (2) 
Developing new courses and programs; (3) Developing new curriculum materials; (4) 
Providing student supports and career development; (5) Recruiting promising students into 
teaching; and (6) Developing exemplary field sites. 

The internal evaluation of the NYCETP was carried out by the Center for Advanced 
Study in Education (CASE) of the Graduate Center of CUNY. During the first three years of 
the project CASE focused on several evaluation activities. These included documentation and 
formative feedback on collaborative workshops and conferences, on-going consultation on 
collaborative goals, and implementation of particular activities such as the workshops. For a 
few individual faculty the evaluators provided technical assistance in conducting a pre-post 
course survey of student attitudes toward mathematics, Views about Sciences Survey, and a 
student survey in biology (see Pape, Tittle, & Flugman, 1999). 

In late spring of the first year the evaluators decided to focus evaluation activities on 
two key NYCETP project goals: (1) faculty development; and (2) intra and intercampus 
collaboration in developing courses and curriculum materials. We developed and carried out 
procedures for NYCETP faculty to conduct cross-campus case studies of courses being revised 
and/or developed by other NYCETP faculty participants (Tittle, Pape, & Flugman, 2000). As 
part of the case study evaluation project, the case study process was formalized to some degree, 
involving an outline for the case study and a peer review form designed to describe and 
evaluate the case-studied courses, as well as other courses revised by faculty in the 
collaborative (Pape & Tittle, 1998). The peer review form (Self-Study Guidelines ) included 
check lists and ratings on whether course documents and curriculum met the collaborative 
student-centered instructional goals, course content goals, course/materials minimum 
expectations, and evidence of effectiveness of goals in mathematics and/or science, including 
student attitudes or other outcomes. A glossary of terms accompanied the Guidelines (Pape & 
Tittle, 1998). These forms drew on earlier reviews of assessment and evaluation of 
mathematics and science teaching reform in classrooms (Tittle & Pape, 1995, 1996) and the 
national standards in mathematics and science. 

The work with the course Self-Study Guidelines was extended to a procedure to 
evaluate student lesson plans prepared by mathematics teacher education students. The 
procedures were tried out in a pilot study (Scheiner & Tittle, 2000), and are described briefly 
below ((EI.B.) . The peer review process in the case studies was used to support the 
Collaborative goals of cross-campus collaboration, cross discipline study, and faculty 
development linked to the instructional process. The case studies dealt with science and 
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mathematics courses that were content focused, and the lesson plan study focuses on education 
students in teaching methods classes and on evaluating their lesson plans using NCTM 
Standards. 

In summary, the evaluation of the NYCETP used activities designed to provide direct 
feedback to project directors and faculty participants about workshops and conferences, about 
course curricula and materials that were developed, and about using national mathematics and 
science standards to evaluate student lesson plans. These activities involved procedures of: 
direct observation; developing, summarizing, and reporting results of “feedback” 
questionnaires; documenting attendance and participation in meetings; attending project 
planning meetings and reporting findings; drawing on and adapting procedures for case studies 
so that faculty could study another faculty member’s course; designing peer review forms for 
faculty to evaluate course materials; and adapting the peer review forms for use with student- 
developed lesson plans. All of these evaluation activities and procedures are included in the 
annual reports for years 1-4, and were submitted with the project annual reports and reports to 
the National Visiting Committee. Sections II and III, below, provide descriptions or examples 
of these activities and findings. Section IV describes the work of the Sixth Year Extension and 
the Three Year Evaluation Proposal. References to project reports and ERIC documents are 
included. 



ERIC 
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II. Documentation and feedback for conferences and workshops 



The CASE annual evaluation reports for the first four years of the project provide 
summaries of the documentation and feedback provided to the project directors based on 
participant responses to questionnaires. Year one (1995-96) provided data based on four 
faculty workshops and two larger conferences that included public school personnel as well as 
faculty. Year 2 (1996-97) included responses to four meetings of the faculty working groups 
and one large conference (held at the New York Hall of Sciences) (Pape, Tittle, & Flugman, 
1999). Year 3 (1997-1998) included several meetings of the curriculum groups and a large 
conference held at the American Museum of Natural History (Tittle, 1998). Year 4 included 
two large conferences, both held at the New York Academy of Sciences: Issues facing 
mathematics and science teacher preparation (winter); and Meeting the new Regents 
requirements (fall) (Tittle, 1999). The Year 4 conferences were a response to changes in the 
New York State procedures and requirements for teacher preparation and certification. (A 
pilot study of a technology survey was also carried out and described in the Year 4 report.) 
Year 5 activities focused on dissemination, and did not involve formal meetings of the working 
groups or a large conference. 

As an example of the type of feedback provided by faculty attending workshops, a 
summary of the suggestions and comments for future workshops was made based on Year 1 
workshops (Pape, Tittle & Flugman, 1999). Participant comments reflected a general need for 
practical workshops, with content that could be readily used by faculty in their attempts to 
revise their own practice. Workshops that presented hands-on experiences were received most 
favorably and many of these sessions provided ideas that faculty felt they could implement in 
their own classrooms. Faculty also requested smaller, more concentrated group discussions of 
issues related to course revisions. Specifically, faculty suggestions for future workshops 
included the following: 

1 . Strategies for college faculty to change instruction to a more inquiry-based or 
problem solving model. 

2. Sessions that allowed ample time to interact with materials and computer 
software, including the internet. 

3. Greater facilitation of the collaborative process between and within campuses 
through joint presentations. 

4. Joint workshops with NYC public school personnel to investigate similarities 
and connections between higher education faculty and public school teachers. 

5. Workshop sessions related to course revision activities, with the sessions 
including in-depth discussions of actual course examples of student work and 
opportunity for questions. 

Other summaries of individual workshops and/or curriculum working groups included similar 
suggestions. The request for faculty development activities was a continuous one. 




III. Evaluation to support faculty development and inter-campus collaboration 



Prominent among the goals of the NYCETP were activities related to developing new 
courses and curriculum materials, as well as rethinking college instruction in line with the 
national standards in mathematics and science. In the spring of Year 1 evaluators reviewed the 
collaborative work to date and decided to attempt to focus the evaluation activities to actively 
promote very targeted NYCEPT goal. These goals were faculty development and intra- and 
inter-campus collaboration in developing courses and curriculum materials. Over the course 
of the next three years we developed several evaluation activities that supported both faculty 
development and inter-campus collaboration. These activities included the case study of 
courses, a related peer review process for course documentation, and a review form for student 
lesson plans developed in teacher education methods courses. 



A. The case studies and peer review process of course documentation 



We developed plans and procedures for cross-campus case studies of courses being 
revised and/or developed by NYCETP participants (Tittle, Pape, & Flugman, 2000). A case 
study strategy was deliberately designed to involve faculty in the same discipline area (i.e., 
science or mathematics) to talk one-on-one with another faculty member about a specific 
course. Although many of the conferences and workshops involved faculty presentations about 
a particular course or curriculum, there was not the detailed analysis of the context of the 
course, the students, and the curriculum that would be involved in a case study approach. 
Further, responses to evaluation feedback forms at these workshops and conferences confirmed 
that these activities provided formal and informal forums to converse about common ideas, 
issues, experiences and concerns. However, the activities left faculty expressing a number of 
needs. These needs included requests for: more information on strategies to change 
instruction; more feedback and guidance on changing course materials; more in-depth 
discussions of actual course examples (including student work); and opportunities to sit in on 
innovative math and inquiry-based courses as well as facilitation of inter- and intra-college 
faculty visits. 

All of these evaluation feedback reports supported the decision to have the evaluation 
activities focus clearly on the goals of faculty development, specific courses, and the cross- 
campus involvement of faculty in a case study process. Our goals were: (1) to focus on key 
courses taken by teacher education students — whether in liberal arts and sciences (A & S) or 
education; (2) to have NYCETP faculty from one campus go to another campus; (3) where 
possible, to involve in each individual case study an A & S faculty member and an education 
faculty member; and (4) where possible, to have the faculty member observe an actual class in 
the course being case studied.. These goals have been meet to varying degrees in the case 
studies conducted over the five years of the project, as discussed below. 
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The case study process. The most frequent use of case studies in evaluation is illustrated 
by projects such as the one carried out by Stake and his colleagues (1993). In their project a 
group of evaluators very experienced in writing case studies in evaluation visited a series of 
NSF-funded projects in teacher education and wrote in-depth descriptions of each project for 
archival purposes. These descriptive documents are often considered “non-traditional” 
program evaluation (Frechtling. 1995), and are also more frequently used now in mixed 
method evaluations (Frechtling & Sharp, 1997). Case studies are valued for providing 
sufficient information that readers can form their own interpretations of the “case” being 
presented. Individual evaluators visit each project or case (or course in NYCEPT) and write a 
case study, much as an individual anthropologist or field-based researcher in sociology would 
do (LeCompte, Millroy, & Preissle, 1992). 

In the context of NYCETP, we formalized the case study to some degree in order to 
assist faculty to focus on aspects of the course that met the NYCETP goals. We drew on 
earlier work (Tittle & Pape, 1996) to develop an outline for the case study. The purpose of the 
outline was to provide guidelines for faculty writing the case studies. The outline included the 
following categories: context, students-target population, faculty background, physical 
facilities, curriculum and materials, instructional methods, student outcomes and assessments, 
faculty roles, cross-discipline and field site collaboration and course revision plans. Both the 
year one outline and a revised outline based on faculty feedback are available in ERIC (Pape 
& Tittle, 1998). 

The intent of the case study process was also to develop baseline reports that provided 
information about the courses before revision, as well as information on faculty practices and 
beliefs about teaching at that time. In the first year of the project the co-principal investigators 
of the six NYCETP campuses were asked to identify one or two courses on each campus for 
detailed documentation. They were also asked to identify faculty on their campuses who teach 
courses similar to those identified for study, to carry out the case studies, that is, to write a 
detailed description following the outline. The case study faculty then visited another campus 
to observe a class and meet with the course instructor to obtain details about the course 
curriculum, materials, instructional methods, student outcomes and assessments. Once the case 
study was written, it was sent to the evaluators, who reproduced copies and distributed them to 
the faculty participants, the NYCETP central office, and one to each campus co-principal 
investigator for the campuses involved in an individual case study. 

Faculty participants in the case studies were given stipends of $750 to write a case study 
and $250 to be case studied. Faculty members who were teaching the courses were responsible 
for meeting with the faculty writer, collecting examples of course materials and student work, 
and clarifying aspects of the course as needed by the writer. 

The case study outcomes and products: The Guidelines. Year one case studies were 
carried out for eight courses and involved 10 faculty, 3 in education and 7 in A & S on the six 
campuses. Three courses were offered in education departments, four in mathematics 
departments, and one in a science department. Year two case studies were carried out for four 
courses and involved 9 faculty (3 in education and 6 in A & S) on four campuses. Two courses 
were in science departments, one was in mathematics, and one in education. Year three case 
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studies involved three courses and 6 faculty ( 1 in education and 5 in A & S) from 5 campuses. 
One course was in each area — education, mathematics, and science. Year 4-5 case studies 
were carried out for 5 courses, with 8 faculty (2 education and 6 A & S) involved. The faculty 
were from 5 campuses and a community college. One course was in an education department 
and four in departments of mathematics, including a community college department of 
mathematics. 

Over the four years of the case studies all of the NYCETP campuses were involved at 
least once, and a community college was involved in the fourth and fifth years. Thirty-three 
faculty members participated across the four years and twenty courses were documented in the 
process. These courses were distributed across the areas of education (6), mathematics ( 10), 
and science (4). As these numbers show, the sciences were not as well represented as 
mathematics. 

Following the first year’s case studies, faculty were interviewed about the case study 
process. Faculty reported that the outline was useful and the interactions had facilitated 
collaboration across campuses and understanding of reform-based teaching and learning, in 
some instances. The in-depth visit on another campus assisted faculty to become clear about 
facilities that were necessary. One faculty member reported that she was better prepared to 
provide a request for space and materials than she had been prior to writing the case study. 
Others reported changes in thinking about course revisions, such as incorporating more 
computer graphics and simulations, evaluation of entrance requirements for courses, increasing 
collaboration among students, and using manipulatives as an integral part of a course, and the 
need for greater coherence between math and math education courses. One faculty member 
interviewed reported the difficulties inherent in collaboratively revising courses (i.e., A & S 
faculty and education faculty). 

The case study documents are the primary outcomes of the case study process, and a 
related, peer review process was recommended and described in year two (Pape and Tittle, 
1998). Although the peer review process was not carried out, a set of guidelines and forms 
developed for the peer review process were used. The NYCETP Guidelines for Self-Study of 
Course Documents/Curriculum was used in two ways. The first was in conjunction with 
faculty workshop/meetings discussing sample course documents and revisions. In this instance 
the Guidelines provided feedback to faculty. The second was with the course case study 
documents, and in this instance the Guidelines served to provide some indication of the fidelity 
of the course to national standards and NYCETP goals. 

The Self-Study Guidelines included check lists and ratings on whether course 
documents/curriculum met the collaborative student-centered instructional goals, course 
content goals, course/materials minimum expectations, and evidence of effectiveness of goals 
in mathematics and/or science, including student attitudes or other outcomes. There were also 
ratings for CETP programmatic goals (e.g., collaborations, alternative assessments, 
partnerships, urban context, and dissemination goals). The Guidelines were accompanied by a 
glossary of terms. Ratings of 1 3 course revision documents were summarized at the end of 
year 3 (Pape & Tittle, 1998). The ratings provide some indication that these courses were 
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more student centered — that is, there was at least some use of inquiry-based approaches, focus 
on deeper understanding, and/or an emphasis on problem solving and critical thinking. 

Case study benefits. The outcomes described above do not adequately convey the 
richness, depth, and impact on faculty of some of the case studies. Qualitative outcomes 
provide another perspective on the benefits of the use of faculty case studies in evaluations. 
The examples here highlight the benefits of faculty case studies both to the individual faculty 
and to evaluators, as well as supporting the project goals as cited above. 

In the 1996 first year case studies there were five faculty in mathematics and 
mathematics education who formed the beginnings of an enthusiastic working group in 
mathematics that met through the next two years of NYCETP activities. The individual 
meetings of pairs of faculty to discuss courses and common problems resulted in 
correspondence between them and sharing of course materials. In 1997 there was a case study 
of an exemplary collaboration between a mathematics faculty member and a high school 
teacher. The course, Sequential (high school) Mathematics from an Advanced Standpoint, was 
offered in the Department of Mathematics and Statistics, and was intended for students 
preparing to be high school mathematics teachers. The mathematics professor collaborated in 
the course development and was a participant observer for the duration of the course. The 
course instructor was the high school teacher who was writing an extensive document on the 
course development, syllabus, sample problems and student responses as part of the 
requirements for a masters degree. One of the formative evaluators visited the class in session, 
facilitated the adaptation of the masters project into a case study, asked the mathematics 
professor to write his substantive reflections on the course, and the evaluator also wrote an 
overview to the two documents. The case study process offered flexibility and the resulting 
documents have also been disseminated outside the NYCETP (NSF National Visiting 
Committee and Queens College). 

In 1998 there were also two exemplary case studies, one in science and one in 
science/mathematics education. The weekly one-hour recitation for General Physics: 
Introductory Course in Mechanics, Heat, and Sound was case-studied by a physics professor 
from another campus. The recitation used Mathematica for a series of computer-based 
exercises with a focus on numerical solutions of physics problems. The case study offered the 
physicist an opportunity to thoughtfully place the use of Mathematica for exercises within 
considerations of physics as a science, traditional and reform-oriented physics education, and 
the goal of creative problem solving by analytical mathematics, potentially supported in the 
recitation exercises by numerical methods. 

The second exemplary case study in 1998 was conducted by a professor in the 
Department of Mathematics and Computer Science, who visited a class and met with the 
education professor who developed the course, Applications of Microcomputers to 
Mathematics and Science Instruction, on another campus. The course is conducted with hands- 
on use of major aspects of computer technology, and a syllabus and web links for the class. 
Assignments included developing web pages, group projects, lesson plans, research paper or 
grant proposal, and used “tool software” as well as other instructional software, and was 
required for undergraduate students in the mathematics/science teaching programs. The course 




12 



is highly praised by the computer science professor, who planned to disseminate information 
about the course/web site to education faculty at his own campus. The case study describes an 
effective integration of technology, instructional theory, and science/mathematics, including 
links with schools. The case study benefited the computer science professor, making clear the 
challenge of NYCETP goals: the course required both extensive knowledge of science and 
computer tools-applications, as well as continually evaluating new web sites and creating links 
to them. The course instructor’s major goal was use of technology for enhancing student 
learning in mathematics or science. The qualitative outcomes of faculty learning and 
deepening understanding of the NYCETP and national standards in science and mathematics 
are clear benefits of using the case study process. These are intangible benefits that appear to 
derive from the faculty’s exposure to other teaching examples and the use of case study writing 
to provide an opportunity to focus and reflect on the teaching and learning processes in 
classrooms similar to their own. 

The major benefits for evaluators are “windows” into faculty course procedures and 
materials, as well as faculty reflections on courses other than their own. Further, the case 
studies provide sufficient detail that the NYCETP Guidelines for Self-Study of Course 
Documents/Curriculum can be used with the case studies. It is possible to make judgements 
about the extent to which courses meet NYCETP goals, as was done with the set of individual 
course documents prepared for the Collaborative. Overall, the use of faculty case studies 
provided benefits to faculty and evaluators, and supported overall NYCETP goals of 
collaboration between campuses and education/liberal A & S faculty. 

Summary and implications of the case studies. The NYCETP formative evaluation has 
been innovative in asking university faculty interested in teaching to be involved in conducting 
case studies. The original evaluation impetus for the case studies was to provide baseline data 
on courses designated for reform, and then to restudy these courses when revisions were 
completed. This was an unrealistic expectation. However, the case studies do include several 
excellent examples of reform courses, although at least two of these course reforms were well 
underway when the collaborative began its first year. As mentioned above, the case studies 
provide sufficient detail for project staff, faculty, and evaluators to assess the fidelity of course 
reform to national standards and goals. As the Year 4 evaluation report notes, the case studies 
also illustrate the slow process of change in higher education and locate some of the 
institutional barriers to enhancing the quality of mathematics education for teachers in 
preparation — e.g., use of adjuncts for courses with many sections, variability in instruction 
among sections, including variability in modeling mathematics instructions for these future 
teachers. 

One of the most positive outcome of the case studies was the cross-campus interaction 
among faculty, in depth, about individual courses. From the perspective of formative 
evaluation the case study process directly supported the NYCETP goals. The use of NYCETP 
faculty participants, particularly in the first year, did contribute directly to faculty improvement 
of their course development efforts. This result, along with somewhat similar work by Muller 
(1998), suggests that evaluators, particularly in the formative stages of projects, can add to 
project outcomes by developing strategies to directly involve participants (here, faculty) in the 
on-going work of evaluation. The extension of the case study outline into the peer review 
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process of course evaluation, and into ratings of lesson plans (below), begins to provide a 
network of evaluation activities that support faculty development and can be transferred to on- 
going project activities if project leadership continues. 

The implications from the evaluator’s perspective are to make an active use of 
evaluation activities to involve program participants in developing and/or refining evaluation 
“tools” or instruments as well as using them. Well-structured, these evaluation activities and 
tools become a way to provide information and feedback for the participant’s own use, as well 
as for evaluation. 



B. The review form for student lesson plans: Pilot study 



In the fifth year of the project evaluation, the NYCETP Guidelines for Self-Study of 
Course Documents/Curriculum were adapted and modified for review of lesson plans prepared 
by students in methods courses in elementary mathematics and/or science. The revised rating 
form was used in a pilot study with a small number of education faculty who were teaching 
methods courses. Again, the purpose of these guidelines for reviewing lesson plans was to 
focus on CETP goals and to provide a method for faculty and students to review their work, in 
this case for lesson plans. 

There were two main purposes of the pilot study: (1) to determine if the peer review 
forms developed for the case studies could be adapted and used by faculty; and (2) to 
determine if a sample of the lesson plans developed in classes in mathematics and science 
education met the goals of the NYCETP, that is, the new standards in science and mathematics 
proposed by the National Council of Teachers of Mathematics (1989; 1991) and the National 
Research Council (1996). 

Faculty were asked to collect student lesson plans and then to rate lesson plans from 
methods classes of other faculty. The full report (Scheiner & Tittle, 2000, and submitted to 
ERIC) describes (1) the procedures followed, (2) summary tables and findings for two sets of 
lesson plans rated by two faculty each, (3) a mini-case studies for four of these lessons, and 
(4) conclusions drawn from the pilot study of ratings of student lesson plans. 

Procedures. In brief, the procedures included describing the project to the project 
directors on the six campuses of the NYCETP and requesting the names of faculty members 
who teach elementary education methodology courses in science and math and who agreed to 
be contacted for participation in a cross-campus project involving peer review of lesson plans. 
Participants would be paid $200 for their services. A list of eight faculty members was 
compiled, four in math and four in science. 

Faculty were asked to select a sample lesson plan, rate another faculty’s student lesson 
plan, and to evaluate/comment on the review form itself. The purpose of this first step was to 
evaluate the form itself. By November 1999, five participating faculty members had sent in a 
sample lesson plan written by one of their students. The collected lesson plans were then 
copied and one lesson plan was sent to each participant, along with an NYCETP lesson plan 
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rating form and a glossary of terms. Five participants read the lesson they received, filled out 
the rating form, made comments on the lesson, and suggested changes for the rating form. 

The rating form was modified and five participants collected consent forms from their 
mathematics or science education methods’ students and sent in a randomly selected sample of 
ten lesson plans from those consenting students (names removed for anonymity). 

Not all faculty completed ratings and the pilot study results are based on two sets of 
mathematics lesson plans, for each of which we had two sets of faculty ratings completed by 
April 2000. The ratings were examined for the extent of agreement between the two raters, 
and to determine if there were disagreements on particular items on the rating form. These 
latter disagreements might indicate that certain items were ill-defined or that it was difficult to 
determine an element’s presence in a lesson plan. 

Findings. The two main purposes of the pilot study were to determine 1) the usefulness 
of the peer review forms for faculty ratings of lesson plans, and 2) the extent to which these 
small samples of lesson plans met NYCETP goals. With respect to the first question the pilot 
study provided evidence that the adapted peer review form was useful, and also that further 
revisions were needed. In particular, there were several items that were not consistently used 
by our small sample of mathematics education faculty, and indications that rater agreement 
varied as a function of the degree to which lesson plans were clearly exemplary or inadequate. 
The lesson plans selected for the case studies helped to clarify possible reasons for rater 
agreement and disagreement. Lesson plans at either extreme were consistently evaluated by 
both raters. Lesson plans that were not at the extremes and were lacking in detail presented an 
ambiguity that led raters to have different interpretations on some items. 

With respect to the second question of lesson plans meeting NYCETP goals, there were 
clearly differences between the two sets of lesson plans from mathematics education classes. 

We used a stringent criterion — receiving an overall quality rating of good or excellent on any 
of the three major categories— to answer this question. Set 1 had only two lesson plans that 
were evaluated by both raters as good (3) or excellent (4) in one of the major categories 
reflecting NYCETP goals (Evidence of Effectiveness Goals, and Course/ Material Minimum 
Expectations). Set 4 had six of the 10 lesson plans that were rated good or excellent in at 
least one major category (A-Student-Centered Instructional Goals). Of these six lesson plans, 
two were rated good or excellent in all three of the major categories for which overall ratings 
were given. Although this is a small sample, it does suggest that the reviews of lesson plans 
could be used to examine the extent to which the lesson plans being developed in mathematics 
education classes were meeting Collaborative goals. 

There are also a number of implications to be drawn from these admittedly limited data. 
Despite the extent of rater agreement described, these is a wide range in both the form and 
substance of lesson plans within these two sets. For a project such as NYCETP which is 
attempting to support the application of national standards in mathematics and science for 
teacher education students, student lesson plans provide an opportunity to incorporate a goals 
and standards-oriented framework for both teacher education faculty and students. 
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In our pilot study, the faculty of the methods classes use different directions for 
writing lesson plans. Other methods faculty undoubtedly have their own directions. A review 
form similar to the one used in the pilot study could be useful in faculty development and 
include practice in rating (as is done with raters of other text, e.g., essays), in order to have 
consistent agreement on the rating categories and definitions in the reviews. Also, it is likely 
that if more detailed review forms were provided to students, the lack of detail and ambiguity 
in lesson plans would be reduced and rater agreement would increase. Further, standards 
would be discussed and incorporated into student lesson plans. And, as was shown by one of 
the case study lesson plans, clarity, focus and meeting standards do not mean longer lesson 
plans. In summary, the pilot study of mathematics education faculty rating student lesson 
plans does suggest the potential uses for such a process in supporting the incorporation of 
standards and NYCETP goals as a valuable component of teacher education programs. 
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IV. Sixth Year Extension 



The no-cost extension of the NYCETP for the 2000-2001 year has enabled three 
activities to be carried out by the Center for Advanced Study in Education: 

- Planning and piloting design of an integrated student data base and tracking 
system; 

— Participation in the NSF-funded University of Minnesota Core CETP data 
collection; and 

-- Development of a proposal for a CETP Institutional Focus summative evaluation 
Study. 

All of these activities are continuations of the evaluation work developed by CASE over the 
five years of the NYCETP project. 

A. An integrated student data base system 

Throughout the five years of the NYCETP project there has been discussion of the 
problem of summative evaluation of the teacher education programs, including NYCETP 
programs. For the City University of New York there have been few resources that could be 
focused on the problems of assessment of teacher education students both during their training 
and once they were employed. New York State, however, has been making steady progress on 
the types of assessments used for qualifying teacher education students in basic skills, as well 
as the subject matter assessments and even videotaping of teachers on the job for provisional 
and “permanent” certification. The work of the state has been extended to the requirements of 
standards for teacher education students in basic competencies, and to placing teacher 
education programs on probation or discontinuing their status as certified programs if these 
standards are not met. 

The New York State assessment program for teachers in preparation and for 
certification has meant that these evaluative assessments can be linked to the institutional data 
bases at each campus and the central office of the University Dean of Teacher Education. In 
effect, a meaningful student data base system can be developed for teacher education 
programs. In addition, the NY state requirement that teacher education programs choose one 
of the national accreditation bodies and receive national certification status has added to the 
efforts to extend a CUNY data base to include status and evaluation of the performance of 
program graduates in their employment as teachers, and as participants in graduate school 
teacher education. The National Council for Accreditation of Teacher Education, the 
accrediting body that will be used by CUNY teacher education programs, will eventually 
require the assessment of program graduates (NCATE 2000 Standards, May 1 1, 2000). 

Thus, both state and national agencies requirements support the planning and pilot work 
that has been started this year for an integrated student data base system. These developments 
were discussed at meetings of the PI s of the NYCETP in the fall of 2000. The NCATE 
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standards include the requirement that the teacher education programs have an assessment 
system and unit evaluations. A meeting was held later (Fall 2000) with CASE staff, the 
University-wide Dean for Teacher Education, and the University-wide Associate Dean for 
Institutional Research. This meeting focused on the NCATE requirements and their 
implications for teacher education programs, institutional research units on each campus, for 
the central offices of institutional research and teacher education, and for evaluations of 
specific initiatives such as NYCETP. Following further work and discussions with CUNY 
campus offices of teacher education, a plan was developed by the CASE evaluation team to 
help in designing the student data base (student tracking system) so that it could accommodate 
the collection of the University of Minnesota NSF core evaluation data for both pre-service 
students and employed teachers. 

Further, both the work for the CORE NSF evaluation and the work on the integrated 
student data base project contributed to the development of the proposal for summative 
evaluation submitted to NSF in March 2001. These contributory linkages are described below. 



B. Core CETP evaluation 

NYCETP participation in the Core Evaluation Study conducted by the University of 
Minnesota involved three colleges. The surveys were administered in 19 classes that had 
participated in reforming their curricula in keeping with NYCETP criteria. Forms A and B 
were randomly administered within class and returned to CASE. Student surveys for 355 
students were sent to the University of Minnesota.. The college student survey had several 
sections: demographic information; student perceptions of science and mathematics; and use 
of instructional strategies in the specific science, mathematics or education course in which the 
student was enrolled and surveyed. Data are reported below separately for forms A and B 
since the items on the forms differed. In particular, instructional strategies questions on form 
A asked if a strategy happened and if it did. How helpful was the strategy (not helpful, 
somewhat helpful, very helpful). Form B asked How often did you. . .? (seldom, occasionally, 
regularly, almost always). 

The questions on the student surveys were linked to several CETP goals and national 
mathematics and science standards (Lawrenz, Huffman, Appeldoom, & Sun, 2001). For 
example, students were asked how often they worked on problems related to real world or 
practical issues, performed investigative activities including data collection, whether they 
completed assessments or assignments that included portfolios and full-length papers, and 
about the use of technology in classes. In interpreting the student responses below, the type of 
courses in which students were enrolled is very relevant. The 19 classes in the NYCETP 
sample includes 18 courses taught in science and mathematics departments (Schools of Liberal 
Arts and Sciences), of which two are cross listed with the School of Education. One course is 
taught in a School of Education. Thus the student responses need to be considered in terms of 
their experiences in mathematics and science courses taught by faculty in the Liberal Arts and 
Sciences. 
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Results . Response percentages were tabulated for all questions in student surveys A 
andB. These results are presented in Appendix A. Table 1 presents percentages for selected 
demographic questions, Tables 2 and 3 for student perceptions regarding mathematics and 
science, and Tables 4 and 5 for instructional strategies. Table 6 presents results for selected 
instructional strategies where students indicated that the strategy was implemented in their 
class. The nine instructional strategies in Table 6 were selected because they are important 
NYCETP and national standards goals and aims for classroom instruction in science and 
mathematics. 

Table 1 demographics show that 70% of the students are in their junior or senior year. 
The majority of the sample, 76% of survey A (n=180) and 78% of Survey B (n=175), indicated 
that they intend to become licensed as a teacher. When asked the question, Why are you taking 
this course? (check all that apply), 78% indicated that, “It is required by my major,” and 64% 
checked, “Meets a general education requirement.” Only 10% reported that they were taking 
the course because they are interested in it. 

The responses to questions on perceptions about mathematics and science are reported 
in Tables 2 and 3. Almost all students, 94%, agreed or strongly agreed with the statement, 
“There are many methods of solving scientific problems.” And, about 80% disagreed or 
strongly disagreed with the statements that suggested that special abilities were required to 
truly understand mathematics or science in math and science classrooms. 

Table 4 presents response percentages, means and ranks for instructional strategies that 
may have been implemented in the courses and, if so, the degree to which they were helpful. 

On average, across courses and strategies, students indicated that instructional strategies were 
implemented about 66% of the time. This figure reflects an average of the response 
percentages in the "Did Happen" categories in Table 4. 

The instructional strategies were rank ordered in terms of the percentages in the “very 
helpful” category, and the highest percentage received a ranking of one. The four highest- 
ranked strategies received ratings ranging from 40% to 48% in the “very helpful” category: 
Writing descriptions of your own reasoning; Basing new information on what students already 
know about the topic; Performing investigative activities that included data collection, analysis 
and various types of representation; and Working on problems related to real world or practical 
issues. The percentages for these data were also examined by adjusting the percent base by 
deleting the number of students who reported this did not happen in their classroom. The 
adjusted percents on degree of helpfulness of the strategy are described for students 
experiencing the strategies (see below and Table 6). 

The results in Table 4 also provide information on strategies reported as not 
implemented in the classroom (“Didn’t Happen”). Data in Table 5 are generally in agreement 
with the data reported in Table 4, and they expand the ratings in terms of frequency of 
occurrence (from Seldom to Almost Always). Notable are the low uses of technology in 
instructional strategies in both Tables 4 and 5. While these data range from about 35% to 49% 
in the “Didn’t Happen” category for Table 4, Table 6 gives a very positive picture of what 
happens when technology is used. 
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For those who reported that technology was used, the overwhelming majority of these 
students (89-95%) rated the uses as Somewhat or Very Helpful for three uses: 

To understand or explore in more depth concepts used in class (89%) 

To learn new information (92%) 

As a tool in investigations to gather and analyze scientific or mathematical data 
(95%) 

Similar positive ratings occur for other major goals of current CETP reform efforts. 
Table 6 presents selected instructional strategy goals where, again, of those who reported that 
the strategy was implemented , over 80% indicated that it was Somewhat or Very Helpful: 

Participating in whole-class discussions during which the instructor talked less than the 
students (82%) 

Writing descriptions of your own reasoning (92%) 

Working on problems related to real world or practical issues (97%) 

Performing investigative activities that included data collection, analysis, and various 
types of representation (95%) 

Designing and making presentations that help you learn class concepts (89%) 

Completing assessments/assignments that included problems with complex solutions 
(87%) 

There were a few instructional strategies in the assessment category (Table 4) that over 
half the sample indicated did not happen: Completing assessments/assignments that included— 
1) portfolios (63%), and 2) full-length papers (56%). 

Overall, the student survey data provide some evidence that CETP goals are being met 
to an encouraging degree, particularly since the sample consists primarily of students in 
courses taught in departments of mathematics and sciences. For example. Table 5 presents 
data that over half of the students (Regularly plus Almost Always) do the following: Write 
descriptions of your own reasoning (58%); Work on problems related to real world or practical 
issues(58%); and Perform investigative activities that included data collection, analysis, and 
various types of representation (54%). About half the students (49%) also complete 
assessments/assignments that include Problems with complex solutions. Similarly 
encouraging are the responses of students who have these experiences. They are very positive 
for many of the CETP goals, as indicated by the data described (above, Table 6). 
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C. NYCETP Summative Evaluation Proposal 



In March 2001 a proposal, A Follow-up Summative Evaluation of the New York 
Collaborative for Excellence in Teacher Preparation, was submitted to the NSF CETP- 
Institutional Focus unit. The research and evaluation project is designed to examine the effects 
of training teachers to provide mathematics and science instruction aligned with newly 
established content and pedagogical standards. Teacher education students on three campuses 
of the NYCETP will be involved. These K-6 students will be tracked longitudinally for three 
years — the last two years of college and the first year of employment The tracking procedures 
will be part of those established following on the NYCETP sixth year extension that involved 
CASE in planning for an integrated student data base and tracking system. Many of the 
assessment procedures will draw on the University of Minnesota CORE Student Survey and 
Teacher Survey. Performance on New York State Teacher Certification examinations in 
mathematics and science will assess teacher candidate knowledge. Actual teacher performance 
in classrooms will be assessed by the University of Minnesota Core Classroom Observation 
Protocol. 

The study will involve 175 students who are taking CETP courses in the three urban 
colleges as preparation for K-6 teaching. These students will take from 2 to 8 CETP reformed 
courses in mathematics/science and teacher education. A comparison group will be 
constructed from students attending the same colleges who have been waived out of CETP 
reformed courses because they are transfer students or because they already have obtained a 
baccalaureate degree. It is estimated that there were be about 30 of these comparison students 
per college, and about 60 CETP students per college for two years of the study. A smaller 
random sample of 40 CETP teachers and 40 waived comparison (non-CETP) teachers will be 
observed in their first year of teaching. 

Specific research questions to be examined in the proposed study include: How does the CETP 
treatment group compare to the non-treatment group on assessments of knowledge, attitudes 
and self-reported teaching strategies? Are teacher education students at different levels of 
academic ability affected differentially as a function of their CETP/non-CETP participation? 

To what extent does the treatment group’ performance in the classroom change over time 
(from student teaching to employment) as a function of CETP involvement compared to the 
non-treatment group? The proposed study is important to contributing to the understanding of 
how new approaches to teacher training (CETP) can meet the pressures of urban school 
environments. Further, the proposed study will empirically model the use of a comprehensive 
teacher education tracking system. 

Overall, the sixth-year extension of the evaluation component of the NYCETP project 
has enabled progress to be made on planning and designing an integrated student data base and 
tracking system that will continue at the City University of New York on an institutional basis. 
The system will encompass all the senior colleges of the City University that offer teacher 
education program. The majority of graduates of these programs enter urban schools, and 
NTCETP will have a continuing impact on the improvement of NYC schools. 
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V. Summary and implications 



The New York for Collaborative for Excellence in Teacher Preparation (NYCETP) 
formally concluded in July 2001. The project included four years of developmental activities, a 
fifth year of dissemination activities, and a sixth year extension to complete dissemination, 
project and evaluation reports, participate in the CORE NSF evaluation data collection, and 
develop a longitudinal research/evaluation proposal of a student cohort at three college 
campuses of the City University of New York. The Center for Advanced Study in Education 
(CASE) was the internal evaluation contractor for NYCEPT, and carried out on-going 
formative evaluation activities during the five years of the project, as well as several of the 
sixth year extension activities (described earlier). 

During years 1-4 the meetings of the Principal Investigator and the co-Principal 
Investigators from the participating colleges were attended and documented by CASE staff. 
Faculty meetings in working groups and conferences were attended and feedback surveys 
carried out and reported. The pattern of activities for the first three years included faculty 
workshops and, in years two and three, faculty curriculum working groups, as well as at least 
one large conference each year. The pattern of attendance for each of these activities was one 
of decreasing attendance with each succeeding year. A distinct exception was the second year 
large conference held at the New York Hall of Science in Queens, on Mathematics and science 
for all: How informal science-rich institutions can contribute to national reform efforts in 
mathematics and science. This conference was attended by 161 public school, university, and 
museum personnel, including students. A somewhat similar pattern held for the case studies 
initiated by the evaluators. The greatest number of faculty and courses were involved in year 
one, in which 8 NYCETP ‘reform’ courses were documented, with 10 participating faculty. 
Year 2 involved four courses and 9 faculty, year 3 involved 3 courses and 6 faculty, and years 
4-5 involved 5 courses and 8 faculty. Faculty in science, mathematics and education were 
involved in all of these case studies, as well as faculty across the participating campuses of the 
collaborative. In all, 33 faculty participated, 20 courses were documented, and the courses 
were distributed across the areas of education (6), mathematics (10), and science (4). 

The NYCETP had many diverse yet related goals intended to enhance the preparation 
of teachers at the six campuses. While many of the course development activities and case 
studies were carried out across campuses, other activities tended to be more involving of 
individual campuses. This was true for activities related to the teaching scholars, providing 
student supports and career development, developing exemplary field sites and involving 
master teachers, and recruiting students. There were three issues of a collaborative 
publication, Connect (1995-spring 1997), and one issue of a newsletter, NYCETP News (fall 
1997, beginning of the third year of the Collaborative). In 1999-2000 there was a web site at 
Brooklyn College, and a publication was prepared, Faces of Reform: Snapshots of exemplary 
MST activities of the New York Collaborative for Excellence in Teacher Preparation. A 
broader dissemination of some of the activities was targeted in another publication, a special 
issue of the Journal of Mathematics and Science: Collaborative Explorations (Vol. 3, 1, Spring 
2000 ). 
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Overall, the NYCETP achieved some of its extensive goals. The core goals of reform 
in mathematics and science courses, and the related education courses should be sustained and 
continue to expand. This will primarily be due not only to NYCEPT but also the context in 
which it was situated. The state of New York has been active in reforming teacher certification 
and programs of teacher education. Requirements for change in certification programs include 
many of the CETP goals of education, as well as requirements for mathematics and science 
faculty working together to improve teacher preparation courses. The NY State Standards for 
K-12 students emphasize many of the same goals as the national standards movements in 
mathematics and science. State assessment programs for teacher education students and 
teacher certification have been implemented and standards set on these assessments. All 
teacher education programs have been required to revise their programs in line with these goals 
and processes. Colleges and universities must submit their programs for re-certification, as 
well as prepare and obtain certification from a national board of certification, such as the 
National Council for Accreditation of Teacher Education. While NYCETP has contributed to 
meeting all of these contextual demands, all of these situational pressures will in turn help to 
ensure that the goals and activities of the NYCEPT continue in some form in the six campuses 
that constituted the Collaborative. 
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APPENDIX A 




Table 1 

Demographics of College Student Surv eys A and B: Response Percentag es 



Survey A Survey B 

(N=180) (N=175) 



Institution: 



College (1) 48.3 47.4 

College (2) 49.5 50.3 

College (3) 2.2 2.3 

Gender: 

Male 17.6 

Female 82.4 

Classification year: 

First year - 6.9 

Sophmore - 22.0 

Junior - 27.7 

Senior - 42.2 

Graduate - .6 

Non-degree/licensure - .6 

Age 

17-21 - 51.0 

22-26 - 29.9 

27-32 - 7.0 

33-37 - 8.3 

38+ - 3.8 

Do you intend to become licensed as a teacher: 

Yes 76.0 78.0 

No 24.0 22.0 
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Page 2 
Table 1 



Survey A 

(N=180) 



c 

Are you already participating in a teacher 
preparation program: 



Yes 


50.6 


No 


49.4 



If you are particpating in a teacher preparation 
program: 




How would you rate the level of quality of 
your courses and practicum experiences? 
(n=97) 




Less than adequate 


6.2 


Adequate 


57.7 


More than adequate 


27.9 


Exceptional 


8.2 


Are the teaching methods used in this class 
similar to those presented in your methods 
courses? (n=97) 




Yes 


- 


No 


- 


Do you have a scholarship provided by a Collaborative 
for Excellence in Teacher Preparation (CETP) or the 
National Science Foundation (NSF)? 




Yes 


1.7 


No 


98.3 



Note: (-) indicates that no data is available for that question. 




Survey B 
(N=175) 



47.0 

53.0 



39.2 

60.8 



1.8 

98.2 



28 



Table 2 

College Student Survey A: Perceptions 
Response Percentages and Means (N=180) 



Please darken one circle for each statement below. 



Strongly Strongly 





Agree 


Agree 


Disagree 


Disagree 


Mean* 


The truth of scientific knowledge is 


beyond doubt. 


6.8 


52.0 


33.3 


7.9 


2.42 


There are many methods of 


solving scientific problems. 


32.4 


62.0 


4.5 


1.1 


1.96 


Truly understanding science in the 
science classroom requires special 
abilities that only some people possess. 


3.4 


15.2 


55.6 


25.8 


3.04 


Truly understanding mathematics in 
the mathematics classroom requires 
special abilities that only some 
people possess. 


4.4 


15.0 


55.6 


25.0 


3.01 



*Note: The mean is derived from responses that are coded from 1 to 4 with "1" representing "strongly agree" and 
H 2" representing "agree", and so on. 
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Table 3 

College Student Survey B: Perceptions 
Response Percentages and Means (N=175) 



You must choose the single best answer-darken one circle only. 



Response 

Percentage 



In deciding whether or not a proposed theory can be accepted, 
scientists will probably make their decision on the basis of: 



a. Whether or not the theory is true. 



4.7 



b. Whether or not the theory can be expressed 
in mathematical form. 



5.2 



c. The evidence supporting the theory and their 
personal ideas. 



22.8 



d. The experimental and observational evidence 

available. 67.3 

*Mean: 3.53 

Most important scientific advances have come about as a 
result of: 

a. The development of new and more scientific sets of 

ideas. 22.5 

b. The interaction of ideas and experiments in the solution 

of problems. 52.7 

c. The dedication of an extraordinary person to the 

investigation of a particular specialty. 4.7 

d. An interaction between a chance observation of a 

new phenomenon with an alert mind. 20. 1 

*Mean: 2.22 



*Note: the mean is derived from responses coded from 1 to 4 with "1" representing 
choice "a" and "2" representing choice "b", and so on. 



Table 4 

Instructional Strategies: Survey A 
Response Percentages,. Means and Ranks 



The following items represent dimensions of instruction that might have been implemented in this class. If particular 
instructional strategies were not implemented, please check "Didn't Happen". If they were implemented, please indicate the 
degree to which the strategies were helpful to you in learning course concepts and content. Darken one 
circle for each item. 



j>?d H appe n 

Didn’t Not Somewhat Very 

Happen Helpful Helpful Helpful Mean a 



Working with other students where the 

whole group gets one grade? 48.9 

Participating in whole-class discussions 

during which the instructor talked 

less than the students? 25 .8 

Writing descriptions of your own 

reasoning? 12.4 

Working on problems related to 

real world or practical issues? 10.2 

Performing investigative activites that 
included data collection, analysis, and 
various types of representation? 23 . 7 

Making connections to to other SMT* 

and non-SMT*fields ? (*science, 

mathematics and technology) 27.7 

Designing and making presentations that 

help you learn class concepts? 45.4 

Evaluating the extent of your own 

learning? 17.7 

Having a voice in decisions about course 27. 1 

activities? 

Basing new information on what students 
already know about the topic? 13.3 

Using student assessment results to 

modify what was taught and how? 28.0 







7.8 


25.0 


18.3 


2.1 


19.0 


13.5 


34.9 


25.8 


2.6 


13.0 


6.8 


41.3 


39.5 


3.1 


4.0 


2.8 


39.0 


48.0 


3.3 


1.0 


4.0 


31.6 


40.7 


2.9 


2.0 


5.6 


38.5 


28.2 


2.7 


9.0 


5.7 


25.9 


23.0 


2.3 


14.0 


5.7 


40.0 


36.6 


3.0 


5.0 


10.2 


29.4 


33.3 


2.7 


7.0 


8.1 


38.7 


39.9 


3.1 


3.0 


8.9 


36.3 


29.8 


2.7 


8.0 
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Page 2 
Table 4 



JD M Jgappj^ 



Didn't Not Somewhat Very 

Happen Helpful Helpful Helpful Mean a R** 



Completing assessments/assignments that 
included: 



problems with complex solutions? 


22.9 


9.7 


40.5 


26.9 


2.7 


11.0 


portfolios? 


62.6 


5.3 


19.5 


12.6 


1.8 


21.0 


multiple choice/short answer items 


37.9 


5.0 


36.2 


20.9 


2.4 


16.0 


full-length papers? 


56.5 


9.4 


18.8 


15.3 


1.9 


20.0 


Using technology, e.g., computers, calculators: 
to practice skills learned in class? 


43.3 


5.0 


31.5 


20.2 


2.3 


18.0 


to understand or explore in more 
depth concepts learned in class? 


37.6 


6.8 


33.1 


22.5 


2.4 


15.0 


to learn new information? 


34.6 


5.1 


26.8 


33.5 


2.6 


6.0 


as a tool in investigations to gather 
and analyze scientific or 
mathematical data? 


37.9 


2.8 


31.6 


27.7 


2.5 


10.0 


as a tool to prepare written reports 
or presentations? 


48.9 


2.2 


22.2 


26.7 


2.3 


12.0 


as a tool for assessment? 


42.2 


5.0 


32.2 


20.6 


2.3 


17.0 


a The mean is derived from responses coded on a 1 to 4 scale with "1" 


representing 


"didn't happen" 


and " 2 ” 


representing 



helpful", and so on 



ranking (R) of " 1 " indicates that this item received the highest percentage of respondents reporting that this strategy was 
implemented and was "very helpful" in this course. A ranking of "2" indicates that this item received that second highest 
percentage of respondents reporting thatthis strategy was implemented and was "very helpful" in this course, and so on. 
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Tables 

Instructional Strategies: Survey B 
Response Percentages, Means and Ranks 



The following items represent dimensions of instruction that might have been implemented in this class. Please respond how 
often you were asked to do each of the items. Darken one circle for each item. 

How often did you? 





Seldom 


Occasionally 


Regularly 


Almost 

Always 


Mean a 


R b 


Work with other students where the 
whole group gets one grade? 


42.9 


32.9 


15.4 


8.8 


1.9 


21.0 


Participate in whole-class discussions 
during which the instructor talked 
less than the students? 


21.3 


41.4 


25.5 


11*8 


2.3 


12.0 


Write descriptions of your own reasoning? 


11.9 


29.8 


35.1 


23.2 


2.7 


6.0 


Work on problems related to real world or 
practical issues? 


10.8 


31.6 


38.0 


19.6 


2.7 


1.0 


Perform investigative activites that 
included data collection, analysis, and 
various types of representation? 


20.3 


25.6 


37.2 


16.9 


2.3 


4.0 


Make connections to to other SMT* and 
non-SMT*fields ? (* science, 
mathematics and technology) 


27.4 


29.8 


29.7 


13.1 


2.3 


10.0 


Design and make presentations that 
help you learn class concepts? 


44.1 


27.3 


19.9 


8.7 


1.9 


17.5 


Evaluate the extent of your own learning? 


21.1 


30.4 


34.5 


14.0 


2.4 


7.0 


Complete assessments/ assignments that 
included: 














problems with complex solutions? 


20.6 


30.0 


37.6 


11.8 


2.4 


2.0 


portfolios? 


56.6 


21.7 


16.3 


5.4 


1.7 


20.0 


multiple choice/short answer items? 


38.0 


24.5 


24.6 


12.9 


2.1 


13.0 


full-length papers? 


44.8 


23.3 


19.0 


12.9 


2.0 


19.0 
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Page 2 
Table 5 

How often did you? 



Almost 

Seldom Occasionally Regularly Always Mean a 



Use technology, e.g., computers, calculators: 



to practice skills learned in class? 


37.5 


32.8 


23.2 


6.5 


2.0 


16.0 


to understand or explore in more 
depth concepts learned in class? 


28.1 


33.6 


26.3 


12.0 


2.2 


11.0 


to learn new information? 


27.6 


26.5 


31.8 


14.1 


1.0 


9.0 


as a tool in investigations to gather 
and analyze scientific or 
mathematical data? 


31.8 


34.1 


23.5 


10.6 


2.1 


14.0 


as a tool to prepare written reports 
or presentations? 


38.0 


21.6 


23.4 


17.0 


2.2 


15.0 


as a tool for assessment? 


44.1 


27.3 


19.9 


8.7 


2.0 


17.5 


How often? 














Did students have a voice in decisions 
about course activities? 


17.0 


31.0 


35.6 


16.4 


2.5 


5.0 


Was new information based on what 
students already know about the topic? 


14.5 


39.5 


32.0 


14.0 


2.5 


8.0 


Were student assessment results used 
to modify what was taught and how? 


21.4 


30.4 


37.5 


10.7 


2.4 


3.0 



a The mean is derived from responses coded on a 1 to 4 scale with "1" representing "seldom" and "2" representing 
"occasionally", and so on. 

b A ranking (R) of H 1 " indicates that this item received the highest percentage of respondents reporting that this strategy was 
implemented "regularly" in this course. A ranking of "2" indicates that this item received that second highest percentage of 
respondents reporting that this strategy was implemented "regularly" in this course, and so on. 
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Table 6 

Selected Instructional Strategies: Survey A 
Response Percentages and Means 



The following items represent dimensions of instruction that might have been implemented in 
this class. If particular instructional strategies were not implemented, please check "Didn't 
Happen". If they were implemented, please indicate the degree to which the strategies were 
helpful to you in learning course concepts and content. Darken one circle for each item. 



Not Somewhat Very 

Helpful Helpful Helpful Mean 3 



Participating in whole-class 
discussions during which the 
instructor talked less than the 

students? 18.2 47.0 34.8 3.2 

(N=132) 

Writing descriptions of your 

own reasoning? 7.7 47.1 45.2 3.4 

(N=155) 

Working on problems related to 

real world or practical issues? 3.1 43.4 53.5 3.5 

(N=159) 

Performing investigative activities 
that included data collection, 
analysis, and various types of 

representation? 5.2 41.5 53.3 3.5 

(N=135) 

Designing and making 
presentations that help you 
leam class concepts? 

(N=95) 



10.5 



47.4 



42.1 



3.3 



Page 2 
Table 6 





Not 

Helpful 


Somewhat 

Helpful 


Very 

Helpful 


Mean 8 


Completing assessments/ 
assignments that included. 










problems with 
complex solutions? 
(N=T35) 


12.6 


52.6 


34.8 


3.2 


Using technology, e.g., computers, 
calculators: 










to understand or explore in more 
depth concepts taught in class? 
(N=lll) 


10.8 


53.2 


36.0 


3.3 


to learn new information? 
(N=117) 


7.7 


41.0 


51.3 


3.4 


as a tool in investigations to 
gather and analyze scientific 
or mathematical data? 


4.5 


51.0 


44.5 


3.4 



(N=110) 



a The mean is derived from responses that are coded on a 2 to 4 scale with "2" representing "not 
helpful, "3" representing "somewhat helpful" and "4" representing "very helpful". 

Note: Response percentages include only those respondents who reported that a particular 
strategy was implemented. 
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